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Executive Summary 

The Special Education Technology Integration Specialist (SETIS) program provides 
professional development for special education teachers to assist them in achieving profi- 
ciency with 21st Century Technology Tools. The program completed its eighth and ninth 
rounds during the 2012-2013 and 2013-2014 school years, training more than 30 special ed- 
ucators as models, coaches, and mentors of technology integration at schools and within class- 
rooms. This study examines SETIS program implementation, use, and impact across three key 
stakeholder groups: SETIS candidates, teacher colleagues, and school administrators. 

Methods 

SETIS candidates were surveyed once using a retrospective pre-post survey conducted 
at the conclusion of the school year. The majority of the SETIS survey items were dedicated 
to measuring pre- to postprogram shifts in four different indices: human capacity, organiza- 
tional capacity, structural capacity, and material capacity. Teacher colleagues and school 
administrators, identified and invited by SETIS due to their close working relationships, par- 
ticipated in pre- and postprogram surveys administered at the beginning and ending of the 
school year. Most of the teacher colleague survey items were grouped into five indices that 
measure (a) designing and developing digital-age learning experiences and assessments, (b) 
modeling digital-age work and learning, (c) engaging in professional growth and leadership, 
(d) promoting and modeling digital citizenship and responsibility, and (e) facilitating and in- 
spiring student learning and creativity. For the purposes of this report, the indices are labeled 
as design, model, engage, promote, and facilitate. The school administrator surveys con- 
tained standalone survey items. 

While prior SETIS evaluation reports examined one cohort only, this study encom- 
passes results from two cohorts: the 2012-2013 cohort and the 2013-2014 cohort. Combining 
cohorts in this study is justified because no changes had been made to the SETIS program or 
its evaluation during their two training periods; further an analysis of the data revealed like 
results for both cohorts. Moreover, the inclusion of both cohorts gives the study larger num- 
bers of surveys and observations to examine, increasing our confidence in the statistical reli- 
ability of our findings. 

Findings 

Evaluation question 1 (EQi)— To what extent does the Technology Integration Spe- 
cialist (TIS) program build the capacity of participating TISs to plan and facilitate (a) teach- 
ing and learning, (b) information access and delivery, and (c) program administration?— is 
primarily addressed through the SETIS survey capacity indices, with supporting evidence 
from other survey items. 

On the whole, the SETIS cohorts perceived substantial increases in their capacity to 
plan and facilitate all of the components in EQi. Each capacity index (human, material, or- 
ganizational, and structural) yielded statistically significant increases in scores from pre to 
post program survey results. Additionally, effect sizes for all indices, with the exception of 
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structural, are considered very large. The strength of these results provides reasonable evi- 
dence that the SETIS program equipped its candidates with the capacity necessary to imple- 
ment technology integration techniques within their schools and classrooms. 

To answer EQ2— To what extent do TISs encounter barriers to successful program 
implementation (e.g., financial, temporal, relational, etc.)?— we look at findings from the 
SETIS survey and the school administrator surveys. 

Overall, the SETIS program throughout the years 2012-2014 encountered relatively 
moderate barriers. From the perspective of the SETIS candidates, a lack of time was their 
largest barrier followed closely by unreceptiveness of teacher colleagues and/ or school admin- 
istrators. Access to technology tools among SETIS candidates was not an issue, however, com- 
puter access for students and internet speed was a concern. The SETIS candidates also 
identified several things they would change about the program; specifically deadlines on as- 
signments, clearer expectations, and increased program organization. While these may or may 
not be barriers to program implementation, they should be taken under consideration. Fi- 
nally, a previous barrier of inadequate awareness about the SETIS candidates and SETIS pro- 
gram among school administrators seems to have been alleviated by programmatic 
adjustments. 

Findings from the SETIS teacher colleague pre- and postprogram surveys, along with 
the SETIS school administrator pre- and postprogram surveys, were used to address EQ3— 
To what extent is the level of technology integration in TIS schools positively impacted 
through participation in the program? 

SETIS candidate activities that led to positive impacts in classrooms were associated 
with improved technology integration in classrooms and curriculum, raised technology 
knowledge among teachers, and enhanced student experiences. Perhaps most importantly, 
open-ended survey responses from both the teacher colleagues and school administrators ev- 
idenced the extent to which the SETIS candidates integrated technology in their schools. Fur- 
ther, data analysis indicates improved technology use among teacher colleagues in the design 
and model indices. 

The fourth evaluation question EQ4— In what ways have school administrators and 
teachers leveraged the TIS and the resources provided by the TIS?— is addressed using many 
of the same findings discussed in EQ3 in addition to items from the SETIS survey. 

Collectively, these data suggest some school administrators and teacher colleagues are 
more adequately leveraging their SETIS candidates than others. Wilcoxon signed-rank tests 
revealed positive increases in technology use among teacher colleagues with statistically sig- 
nificant increases in the design and model indices. On the other hand, Wilcoxon signed-rank 
tests indicated statistically significant reduction when comparing school administrators’ pre- 
program survey intentions to the postprogram survey utilization of their SETIS candidates. 
Yet, despite these data, open-ended survey results among the teacher colleagues and school 
administrators portrayed a high level of satisfaction with the accomplishments achieved by 
working with a SETIS candidate. 

The final evaluation question EQ5 —What impact has the TIS program had on stu- 
dents’ technology literacy in participating schools?— relies upon the premise that successful 
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program implementation along with efficient SETIS and technology resource use leads to 
higher teacher, administrator, and student technology literacy levels. To address EQ5, we 
draw insight from the teacher colleague pre- and postprogram surveys and the school admin- 
istrator pre- and postprogram surveys. 

Survey results give evidence that both teacher colleagues and school administrators 
perceived a rise of technology integration in classrooms and curriculum. Further, the majority 
of school administrators reported they had observed increased student engagement as a result 
of technology integration efforts by their SETIS candidate. Analysis of the teacher colleague 
technology use indices showed statistically significant increases in the design and model indi- 
ces. This shift in behavior indicates that, as a result of the SETIS program, teacher colleagues 
who responded to the survey are practicing technology integration techniques and engaging 
their students using technology resources provided by their SETIS candidate. 

Limitations of study 

When interpreting results in any study, it is important to consider inherent limitations 
that may skew findings. Surveys that rely upon self-reported information always have a risk 
of response bias; respondents may exaggerate or underestimate, may have accurate recall dif- 
ficulties, and may report information they perceive as socially acceptable. This study relies 
upon two types of self-reported surveys: a retrospective pre-post survey and two traditional 
pre-post surveys. There are benefits and drawbacks to each type of survey. 

Retrospective pre-post surveys are convenient because they occur once (improving re- 
sponse rates) and the pre-post data are matched at the individual participant level. Some re- 
search postulates that response-shift bias (a type of bias created from presurvey 
overestimation and postsurvey underestimation of one’s knowledge, skills, and/ or behaviors) 
is alleviated through using retrospective pre-post surveys (Moore & Tananis, 2009). Con- 
versely, other research argues traditional pretest-posttest types of surveys result in less biased 
program effectiveness estimates. In a 2011 study, Nimon, Zigarmi, and Allen discovered in- 
flated effect sizes among retrospective pre-post survey items. According to their findings, sur- 
veys with before-and- after items presented side by side may introduce types of bias including 
theories of change, self-presentation, and/or effort justification. Further, to resolve the issues 
of both response-shift bias and exaggerated effect sizes Nimon and colleagues (2011) recom- 
mend administering traditional pre-post surveys with a retrospective pre-post survey. While 
this solution would alleviate some bias, it was not possible to initiate in this study. 

Previous SETIS program evaluation studies exhibited a limitation of small sample 
sizes. This study, in part due to combining the cohorts from 2012-2013 and 2013-2014, does 
not have this limitation; the sample sizes of all three stakeholder groups are adequate for us 
to draw general conclusions about the groups with a certain level of confidence. 

Determining response rates among teacher colleagues and school administrators re- 
mains impossible due to the fact that the SETIS candidates invite an unknown number of 
teachers and administrators to participate in the pre-post surveys. However, the inability to 
match pre- and postprogram survey results among the teacher colleagues and school admin- 
istrators has been addressed by adjusting the study’s methodology. Matching individual pre- 
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and postprogram survey responses allowed us to measure with more confidence shifts in 
knowledge, skills, and behaviors at individual and group levels. 

Conclusion and Recommendations 

In general, survey data from all three SETIS stakeholder groups suggest the 2012- 
2014 SETIS programs were mostly successful. The greatest impact was observed in capacity 
building among the SETIS candidates. The four indices of capacity building— human, mate- 
rial, organizational, and structural— all yielded statistically significant and practically im- 
portant increases from pre- to postprogram survey scores. Among teacher colleagues, the use 
of Wilcoxon signed-rank tests revealed statistically significant increases in technology use for 
the model and design indices. While Wilcoxon signed-rank tests also resulted in statistically 
significant differences between the school administrator pre- and postprogram surveys, the 
differences uncovered a disparity between school administrators’ higher preprogram survey 
intentions to use SETIS candidates’ skills and lower postprogram survey responses about their 
actual utilization. Despite this lower-than-expected leveraging of SETIS resources, other sur- 
vey data indicated successes resulting from the SETIS program as noted above. 

Study results also suggest that SETIS program staff have adopted previous evaluation 
study recommendations. Methodology adjustments allow pre- and postprogram survey 
matching of teacher colleague and school administrator respondents. Also, awareness of 
SETIS candidates and their services increased among school administrators. 

Even with these promising program adjustments, there are several recommendations 
that may further improve the implementation, utilization, and impact of the SETIS program. 
These include holding more face-to-face meetings, promoting scheduling that allows teachers 
and SETIS candidates time to cocreate technology-integrated lesson plans, and encouraging 
further collaboration between SETIS candidates and their school administrators to better lev- 
erage the SETIS candidates and their resources. 
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Introduction 


The Special Education Technology Integration Specialist (SETIS) program completed 
its eighth and ninth rounds during the 2012-2013 and 2013-2014 school years. As an expan- 
sion of the Technology Integration Specialist (TIS) program, the SETIS program is designed 
to increase the professional capacity of selected special educators, SETIS candidates, to act as 
models, coaches, and mentors to fellow teachers so that they can fully use statewide technol- 
ogy resources. In addition to training, the program provides SETIS candidates with tools and 
resources to assist their schools in integrating and using technology to meet West Virginia’s 
Next Generation content standards and objectives. SETIS candidates also work with school 
administrators to implement county and school technology plans. 

This evaluation study was undertaken at the request of the West Virginia Department 
of Education (WVDE) Office of Special Education as were earlier studies of the SETIS program 
for the years 2010-2011 and 2011-2012. 1 Similar to prior studies, this study examines the 
implementation, use, and impact of the SETIS program for school years 2012 through 2014. 
Unlike earlier studies, this one examines any observable shifts in SETIS candidates’ use and 
impact. The addition of this dimension to the study is possible because, while earlier studies 
lacked a mechanism for matching pre- and postprogram survey responses of teacher col- 
leagues and school administrators, we modified our methods to make such matching possible. 

Five overarching evaluation questions guided our examination of the SETIS program: 

EQi. To what extent does the TIS program build the capacity of participating TISs to plan 
and facilitate (a) teaching and learning, (b) information access and delivery, and (c) 
program administration? 

EQ2. To what extent do TISs encounter barriers to successful program implementation 
(e.g., financial, temporal, relational, etc.)? 

EQ3. To what extent is the level of technology integration in TIS schools positively im- 
pacted through participation in the program? 

EQ4. In what ways have school administrators and teachers leveraged the TIS and the 
resources provided by the TIS? 

EQ5. What impact has the TIS program had on students’ technology literacy in partici- 
pating schools? 

The study addresses the evaluation questions using surveys of all three major stake- 
holder groups in the SETIS program: SETIS candidates, teacher colleagues, and school ad- 
ministrators. 


1 See previous reports: http : / /wvde. state. wv.us /research/reports2Q 12 / WVSpecialEducation- 
TISProgramAdministratorsReport2Qi2.pdf: http://wvde.state.wv.us/research/reports2012/WVSpe- 
cialEducationTISProgramTeachersReport2Qi2.pdf: 

http://wvde.state.wv.us/research/rep0rts2013/WVSETISPr0gram2011-2012Evaluati0nRep0rt.pdf 
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Methods 

Beyond one key improvement to the research design— incorporating a method to track 
and match pre- and postprogram survey responses of teacher colleagues and school adminis- 
trators— no other changes were made to the evaluation of the SETIS program. Akin to the last 
evaluation study (Stohr, 2013), 2 we examined the same three stakeholder groups: (a) SETIS 
candidates, defined as special education teachers who were accepted into the SETIS program 
and received specialized training; (b) teacher colleagues, defined as teachers who cotaught 
with SETIS candidates or with whom SETIS candidates believed they had the greatest influ- 
ence regarding technology use; and (c) school administrators, defined as principals, assistant 
principals, or other teachers who work in an administrative capacity in the schools where 
SETIS candidates served. Additionally, we employed the same survey tools: pre- and postpro- 
gram surveys administered to teacher colleagues and school administrators at the beginning 
and conclusion of the school year, and a retrospective pre-post survey administered to the 
SETIS candidates at the end of the school year. All surveys used a combination of multiple 
choice and open-ended items. 

Participant recruitment also mirrored methods used in prior years. Through the Office 
of Special Education, the coordinating office for the SETIS program, all SETIS candidates 
were invited to participate in the retrospective survey. The responsibility to identify and invite 
teacher colleagues and school administrators to the preprogram surveys was given to the 
SETIS candidates. The SETIS candidates were provided hyperlinks to the preprogram surveys 
for their teacher colleagues and school administrators to complete. Via use of our secure 
online survey platform, those teacher colleagues and school administrators who opted to par- 
ticipate in the preprogram surveys were contacted at the end of the school year and asked to 
complete the postprogram survey. Thus, we were able to address a prior study limitation by 
matching pre- and postprogram teacher colleague and school administrator responses. 
Matching pre- and postprogram survey data increases the precision and power of our tools 
because we can observe actual shifts in individual responses, which in turn allows us to have 
greater confidence in any changes observed in aggregate among the groups of teacher col- 
leagues and school administrators. 

As previously mentioned, the survey instruments employed in this evaluation study 
were the same ones described in the most recent previous SETIS evaluation report (Stohr, 
2013). Please refer to that report for further details on the surveys. The following are brief 
descriptions of the surveys used with the three stakeholder groups: 

• SETIS candidate retrospective pre-post survey. These surveys used both multiple 
choice (5-point Likert-type scale of disagreement/ agreement) and open-ended 
items to gather self-reported data by asking participants about their opinions, be- 
haviors, knowledge, etc., before participating in the program and after participat- 
ing in the program. The majority of survey items were dedicated to measuring pre- 
to postprogram shifts in four different indices: human capacity, organizational 
capacity, structural capacity, and material capacity. All capacity indices were 

2 This report is available at http://wvde.state.wv.us/research/reports2013/WVSETISPro- 
gram2Qii-20i2EvaluationReport.pdf . 
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based on definitions found in Determining Capacity within Systemic Educational 
Reform (Century, 1999). The open-ended items asked participants to share their 
viewpoints on the SETIS program, including the components they found most val- 
uable, suggestions for program changes, and barriers encountered throughout the 
year. 

• Teacher colleague pre- and postprogram surveys. Survey items were both multi- 
ple choice (including a 4-point Likert-type scale of frequency) and open-ended. 
Beyond basic demographic information, teachers were asked to self-report their 
technology practices, with the preprogram survey capturing practices prior to 
working with a SETIS candidate, and the postprogram survey measuring shifts in 
those practices as a result of working with a SETIS candidate. Based on the TIS 
Professional Development Models survey items were grouped into five indices (In- 
ternational Society for Technology in Education, 2008), including (a) designing 
and developing digital-age learning experiences and assessments, (b) modeling 
digital-age work and learning, (c) engaging in professional growth and leadership, 
(d) promoting and modeling digital citizenship and responsibility, and (e) facili- 
tating and inspiring student learning and creativity. For the purposes of this report, 
the indices are labeled as design, model, engage, promote, and facilitate. Open- 
ended questions allowed respondents to provide a descriptive account of the roles 
of SETIS candidates, what the respondent hoped to accomplish (preprogram sur- 
vey) or had accomplished (postprogram survey) by working with a SETIS candi- 
date. 

• School administrator pre- and postprogram surveys. These short surveys com- 
prised multiple choice (including 5-point Likert-type scales of likelihood and 
agreement) and open-ended items. The preprogram survey asked school adminis- 
trators how they anticipated utilizing SETIS candidate services, while the postpro- 
gram survey asked administrators to rate how they actually did use SETIS 
candidate services throughout the school year. Additionally, the postprogram sur- 
vey included one item asking if administrators thought the presence of a SETIS in 
their school could be linked to improved student engagement in curricular activi- 
ties via increased use of technology and technological tools. 

While prior SETIS evaluation reports examined one cohort only, this study encom- 
passes results from two cohorts: the 2012-2013 cohort and the 2013-2014 cohort. Combining 
cohorts in this study is justified because no changes had been made to the SETIS program or 
its evaluation during their two training periods; further an analysis of the data revealed like 
results for both cohorts. Moreover, the inclusion of both cohorts gives the study larger num- 
bers of surveys and observations to examine, increasing our confidence in the statistical reli- 
ability of our findings. 


Data Analysis 

Both quantitative and qualitative analysis techniques were used to examine the data 
collected for this study. To analyze quantitative survey items (i.e., multiple choice questions), 
we used SAS 9.2 to produce descriptive statistics including frequencies (i.e., percentages), and 
to calculate and interpret measures of central tendency and dispersion (i.e., means and stand- 
ard deviations). Statistical significance was established by conducting the Wilcoxon signed- 
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rank test using SPSS. The Wilcoxon signed-rank test is a nonparametric statistical hypothesis 
test that may be used when comparing matched samples to assess whether the mean ranks 
differ; it is often used as an alternative to the paired-samples t test when the data cannot be 
assumed to be normally distributed (Lowry, 1999 -2012). The Wilcoxon signed-rank test ex- 
amines the sizes of the differences between two values, such as pretest and posttest data. With 
this statistical test, the null hypothesis is that the median difference between pairs of obser- 
vations is zero (Lowry, 1999 - 2012). The null hypothesis is rejected if the test statistic (a z 
score in this case) yields a p value of 0.05 or less. A p value of 0.05 or less indicates a 95% 
probability that the survey results were not observed due to chance. 

Additionally, effect sizes using Cohen’s d were calculated. As indicated by its name, 
effect size measures the magnitude of difference in the mean scores between two measure- 
ments, in this study the pre- and postprogram surveys. Further, as a measure of strength be- 
tween the differences of two mean scores, effect sizes are not affected by sample size. Cohen 
(1988) classified effect sizes into three categories; small (0.20), medium (0.50), and large 
(0.80). Later the classification of very large (1.30) was added by Rosenthal (1996) and ac- 
cepted into use. For the purposes of this study, any effect size lower than 0.20 is considered 
negligible. Effect sizes were calculated for statistically significant and near-significant survey 
results only; if a result did not approach statistical significance, then further analyses were not 
warranted. 

Qualitative responses (i.e., responses to open-ended questions) were read, reread, or- 
ganized, and coded according to broad themes, taking into consideration that respondents 
sometimes mentioned more than one theme when responding to a question. Although we pri- 
marily discuss the results from matched comments in the pre- and postprogram surveys, there 
are instances when unmatched comments are considered important or impactful to the SETIS 
program. Therefore, some of the unmatched comments are included in the analysis and dis- 
cussion of the research study findings. The Results section includes tables comparing pre- and 
postprogram survey results and descriptions of the themes. The qualitative data included in 
each of the three surveys (SETIS candidate, teacher colleague, and school administrator) in 
this study were designed to enrich our data by providing respondents the opportunity to an- 
swer questions using their own words. 

Results 

Evaluation study results are presented by stakeholder group type. First we will review 
results from the special education technology integration specialist (SETIS) candidate retro- 
spective pre-post survey, followed by the teacher colleague pre- and postprogram surveys and 
the school administrator pre- and postprogram surveys. 

SETIS Candidates Survey 

A total of 31 SETIS candidates responded to the surveys during the 2012-2013 and 
2013-2014 school years (19 and 12 candidates respectively). These respondents came from 21 
counties across the state. Further, while all programmatic levels were represented, high 
schools and elementary schools were most frequently identified as the SETIS candidates’ work 


4 I The West Virginia Special Education Technology Integration Specialist (SETIS) Program 


Results 


setting. The majority of the candidates reported their role as being a classroom teacher. When 
asked if, to their knowledge, their school currently employed any other individual who had 
completed or is currently completing the Technology Integration Specialist (TIS) program, 
just under 70% indicated yes. Additionally, over 70% of the SETIS candidates indicated they 
had conducted formal staff development related to technology integration at their school 
within the past year. 

Building capacity indices 

In response to EQi— To what extent does the TIS program build the capacity of par- 
ticipating TISs to plan and facilitate (a) teaching and learning, (b) information access and 
delivery, and (c) program administration?— the SETIS candidate survey has sets of items 
that when aggregated create four capacity indices : human, organizational, structural, and 
material. Each of the four capacity sections prompted SETIS candidates to retrospectively 
consider their level of disagreement/ agreement with a statement before (pre) and after (post) 
participating in the TIS program. Response options were based on a 5-point Likert-type scale, 
ranging from a low of 1 ( strongly disagree ) to a high of 5 ( strongly agree). 

Human capacity, the first index in the survey, consists of 17 items measuring the intel- 
lectual proficiency and will of the SETIS candidates’ themselves. To see what, if any, change 
SETIS candidates perceived in their own human capacity over the course of the program, a 
pre- and postprogram mean was calculated for each item (see Appendix A, Table 1, page 23). 
The items, consisting primarily of statements about understanding, use, and integration, each 
revealed statistically significant increases in the median difference between pairs of ratings. 

The second index, organizational capacity, is composed of five survey items centered 
on the interaction, collaboration, and communication among individuals within the SETIS 
system (such as other teachers and school administrators). All items in this index yielded sta- 
tistically significant increases in median difference between pairs of ratings (see Appendix A, 
Table 2, page 25). 

Structural capacity, the third index, includes 17 survey items designed to capture con- 
textual elements such as policies, procedures, and formalized practices— that is, elements that 
are part of the SETIS program, but have been set in place independently of the program. Each 
item in this index also produced statistically significant increases in the median difference 
between pairs of observations (see Appendix A, Table 3, page 26). 

The final index, material capacity, measures fiscal and/or other material supports 
available to people (e.g. SETIS candidates) within a system (e.g. the SETIS program). The ma- 
terial capacity index comprises five items focused on technology resources; once more all 
items produced statistically significant increases in the median difference between pairs of 
observations (see Appendix A, Table 4, page 28). 
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To measure perceived overall growth in capacity resulting from participation in the 
SETIS program, the items in each of the indices were averaged together to create standardized 
capacity indices. Illustrated in Figure l, large increases were observed in all of the indices from 
pre- to post-SETIS program involvement. The human capacity index showed the most dra- 
matic increase, 


rising 

from a preprogram 
mean score of 2.97 to a 
postprogram mean of 
4.44. Conversely, while 
still statistically signifi- 
cant, the structural ca- 
pacity index exhibited 
the smallest increase. 
Additionally Wilcoxon 
signed-rank tests were 
performed on each item 
as well as the standard- 
ized capacity indices; all 
differences were statisti- 
cally significant (see Ap- 
pendix A, Table 5, page 
28). 
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Figure 1. SETIS Candidates' Retrospective Pre-Post Assessment of Growth by 
Capacity Type 

Among all capacity indices statistical testing proved the results were 
extremely significant. Each index yielded a p value of <.0001, supporting a 
high level of confidence that the results represent a true shift and are not 
caused by chance. 


Effect sizes were 
computed to measure 
the magnitude of change 
retrospectively reported in the pre-post survey indices. Figure 2 illustrates the effect sizes ob- 
served for the capacity indices (see individual item results, Appendix A, Tables 1-4). The re- 
sults show very large effect sizes for the human, organizational, and material indices. The 
structural index exhibited the smallest effect size, 0.7, which is considered to be a medium 
effect. 
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Figure 2. SETIS Candidates' Retrospective Pre-Post Assessment of Growth: Effect Size by Capacity 


Index 

Effect sizes between 0.80 and 1.29 are interpreted to be large; those above 1.30 are considered very 
laree. 
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Open-ended items 

There were five essay-style, open-ended items on the SETIS candidate survey. These 
questions were meant to capture qualitative information about the SETIS program and enrich 
the quantitative data discussed above. Most SETIS candidate survey participants responded 
to all of the open-ended items, producing a response rate ranging from 90% to 93% per item. 

The first open-ended item asked participants to identity what they felt were the most 
valuable aspects of the program. During analysis, several themes emerged. The most prevalent 
theme was learning about technology resources/tools, with 17 of the 29 noting this theme. 
Two other common themes were collaborating with other teachers/professional learning 
communities (seven of 29) and practice using new technology tools (five of 29). For a break- 
down of all categories, refer to Appendix A, Table 6, page 29. 

Next, SETIS candidates were asked to describe any changes they would suggest for the 
program. Four major themes were identified during analysis: (a) program/content change, 
(b) time change, (c) meeting change, and (d) no change (see Appendix A, Table 7, page 29). 
Of the 28 responses received, 13 fell into the program/content changes theme. Many of these 
suggestions revolved around the desire for deadlines on assignments, clearer expectations, 
and increased program organization. Six SETIS candidates indicated they would not change 
any aspect of the program, and five stated they would like to see meeting changes in the form 
of more face-to-face meetings. 

The third open-ended item asked survey participants to name school structures that 
supported the TIS program and teacher collaboration. The most commonly named support 
was administrative (10 of 29), followed by none (eight of 29), common planning time (seven 
of 29), and professional learning communities (six of 29). See Appendix A, Table 8, page 29 
for the full list of identified supporting structures. 

The fourth item asked SETIS candidates to discuss the ways they worked with the ad- 
ministration at their school to address technology-related issues or concerns throughout the 
school year. While the responses to this question were varied, the three most common themes 
were (a) providing specific assistance (eight of 29), (b) discussing resource/technology needs 
(five of 29), and (c) other (five of 29). Notably, only two SETIS candidates indicated that they 
had not worked with their administrator to address school-level, technology-related issues or 
concerns. Refer to Appendix A, Table 9, page 30 to see all themes. 

Finally, the last open-ended item asked SETIS candidates to list any barriers they may 
have encountered when implementing what they learned as part of the SETIS program. Of the 
seven themes identified during analysis, a lack of enough time was the most common barrier 
(nine of 29), followed by receptiveness of teacher colleagues/school administrators (seven of 
29), access to and quality of technology 4 (five of 29), and none (five of 29). See Appendix A, 
Table 10, page 30 for a full description. 


4 This category includes a lack of computer access for students and slow internet connectivity. 
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Teacher Colleague Pre- and Postprogram Surveys 

A total of 51 teacher colleagues responded to both the pre- and postprogram teacher 
surveys (45 of which came from the 2012-2013 cohort and six from the 2013-2014 cohort). 
If a teacher colleague completed one survey but not the other, those responses were excluded 
from analysis. The ability to pair an individual’s preprogram survey data to their postprogram 
survey data allowed us to measure actual shifts in attitudes and beliefs from one point in time 
to another. Representing 16 counties, the majority of the responding teacher colleagues iden- 
tified themselves as general education teachers, approximately 40% of whom reported having 
16 or more years of experience in the field of education. Further, survey results indicate some 
of the teacher colleagues cotaught with a SETIS candidate, while other responding teachers 
worked within the same school as the SETIS candidate. 

Technology use indices 

The teacher colleague pre- and postprogram surveys were predominately dedicated to 
survey items inquiring about technology use among the SETIS coteachers and teacher col- 
leagues. These items were designed to address EQ4, In what ways have school administra- 
tors and teachers leveraged the TIS and the resources provided by the TIS? Each item asks 
about the frequency (based on a 4-point Likert type scale where 1 = not usually, 2 = some- 
times, 3 = often, and 4 = almost always ) with which teachers employ various teaching strate- 
gies, categorized into the following indices: facilitate, design , model, promote , and engage. 

Designed to capture the level of technology teacher colleagues use to facilitate and in- 
spire student learning and creativity, the facilitate index consists of five items. Most of the 
facilitate item means rose slightly from pre- to postprogram surveys. However, Wilcoxon 
signed-rank tests did not yield statistically significant results (Table 11, page 31). The next in- 
dex, design, consists of five items measuring thoughtful planning and implementation of tech- 
nical assistance and related activities. Mean scores rose for all design items from preprogram 
levels to postprogram levels, but no results were statistically significant (Table 12, page 32). 
The third index, model, comprises four items intended to measure teacher colleague modeling 
of digital-age work and learning techniques/behaviors. All model item mean scores rose from 
pre- to postprogram survey data and one item (7 use digital resources and tools to communi- 
cate with my peers ) showed statistically significant results with a small effect size (Table 13, 
page 33). After the model index, we have the three-item promote index, which assesses 
teacher-colleague promotion and modeling of digital citizenship and responsibility. Among 
these items, pre-post means stayed the same for one item and the other two rose slightly, but 
did not approach statistical significance (Table 14, page 34). The last index, engage, measured 
professional development, growth, and leadership activities of teacher colleagues. Mean 
scores for all of the four items in this index rose from pre- to postprogram surveys and one 
item (I model and teach other educators to use digital tools and resources to promote student 
achievement and learning ) proved to be statistically significant with a small effect size (Table 
15, page 35). 
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After examining the individual items, we performed statistical analysis on the aggre- 
gate indices scores. Figure 3 illustrates the shift in pre- and postprogram survey mean scores 
for each of the technology use indices. Each index demonstrated an increase in mean score 
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program surveys. 
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Open-ended items 

Teacher col- 
leagues were asked 
to respond to three 
essay-style open- 
ended questions to 

provide qualitative Figure 3. Comparison of Teacher Pre- and Postprogram Survey Mean Scale Scores 
data for this study. b V Technology Use Index 

Analyzing com- The two indices that produced statistically significant results, design and model, 
ments from both y' e ^ed small effect sizes. 

teacher colleagues who provided responses in both the pre- and postprogram surveys, we were 
able to discern what, if any, changes occurred in attitudes and beliefs regarding the SETIS 
candidates and SETIS program. 


Specifically, for the first of these open-ended items, we looked for any shifts in teacher 
colleagues’ perceptions of the role of the SETIS candidate. Of the 27 paired pre-post responses, 
seven indicated no change in their view of the SETIS role, 13 articulated more specific roles 
from pre- to postprogram, one wrote a less specific role definition, two described neutral shifts 
in their beliefs about the role of the SETIS candidate, and four expressed a negative shift in 
their perception of the SETIS candidate role. 


The second open-ended item in the surveys asked teacher colleagues what they hoped 
to accomplish by working with a SETIS (preprogram survey) and what they actually accom- 
plished (postprogram survey). Again, analyzing matched pre-post survey responses (n = 27) 
we discovered that 12 teacher colleagues felt their SETIS candidate met expectations, two re- 
ported their SETIS candidate exceeded expectations, five indicated their SETIS candidate 
made other, unexpected accomplishments, four reported their SETIS candidate did not meet 
expectations, and four teacher colleagues provided comments that were categorized as other. 


The third and final open-ended item afforded teacher colleagues the opportunity to 
provide any additional comments they wished to share about the SETIS program. Of the 
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matched pairs (n = 5) all comments were positive, typically indicating in the preprogram com- 
ments enthusiasm about learning more about the SETIS program and technology integration 
and expressing satisfaction with their SETIS candidate and the SETIS program in the post- 
program survey. Also noteworthy, of the unmatched comments, most teacher-colleague pre- 
program survey comments conveyed interest and enthusiasm about the SETIS program. The 
unmatched postprogram survey comments were a mixture of positive and negative remarks, 
with two teacher colleagues stating they were unfamiliar with the SETIS program. 

School Administrator Pre- and Postprogram Surveys 

In total, 33 school administrators participated in both the school administrator pre- 
and postprogram surveys (25 in the 2012-2013 cohort and eight in the 2013-2014 cohort). 
They represented 17 different school districts. Just over 60% of the administrators identified 
their role as principal, while 24% identified themselves as teachers with administrative duties, 
and 15% reported their role as vice principal. Over 90% of the school administrators reported 
that they were aware a SETIS candidate would be in their school for the upcoming school year 
(this item was asked in the preprogram survey only). 

Use of SETIS candidates 

The administrator pre and post surveys focused primarily on answering EQ4: In what 
ways have school administrators and teachers leveraged the TIS and the resources provided 
by the TIS? These short surveys consist of six (preprogram survey) and seven (postprogram 
survey) multiple-choice statements and three open-ended essay style items. Unlike the other 
stakeholder surveys, the school administrator surveys contain stand-alone items only (i.e. the 
items are not combined to create an index). 

The administrator preprogram survey multiple-choice items are phrased to ask ad- 
ministrators how they intend to use their SETIS candidate, while the postprogram survey asks 
school administrators to report how they actually did utilize their SETIS candidate during the 
course of the school year. The preprogram survey answer choices use a 5-point Likert-type 
scale of likelihood (with 1 equaling unlikely and 5 equaling likely). The postprogram survey 
choices are also based on a 5-point scale, but this scale measures agreement (where 1 equals 
untrue and 5 true). Depicted in Figure 5, the intention was high among school administrators 
to utilize their SETIS candidate as a resource for their school and classrooms. However, post- 
program survey results indicate administrators did not leverage their SETIS candidate to the 
extent they had originally planned. In fact, Wilcoxon signed-rank tests revealed statistically 
significant differences from the preprogram survey intention to the postprogram survey utili- 
zation of the SETIS by school administrators among all items (see Appendix C, Table 17, page 
37). It should be noted the disparity between the intention to utilize and actually utilizing the 
SETIS candidates may be attributed to a number of factors, including unrealistic expectations 
or unforeseen barriers among school administrators responding to the preprogram survey. 
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Figure 5. School Administrator Expected and Actual SETIS Candidate Use by Survey Item 

The effect sizes for these six statistically significant items ranged from small (Contribute to strategic plan) to 

very large (Identify digital resources and Analyze student data). 

Despite the above results showing areas where school administrators could improve 
how they leverage their SETIS candidates, the majority of school administrators reported in- 
creased student engagement (among all students) as a result of their SETIS candidates’ pres- 
ence in their schools. When asked if they had observed increased student engagement in 
curricular activities via in- 
creased technology use that 
could be attributed to the 
presence of a SETIS candi- 
date, most school adminis- 
trators indicated they had 
(see Figure 4). Of the 27 
school administrators re- 
sponding to this survey 
item, just over 50% chose 
the strongest level of agree- 
ment (on a scale from 1 to 5 
where 1 equals untrue and 
5 equals true). Also note- 
worthy, no administrators 
i i ... £ Figure 4. School Administrator Postprogram Survey: Increased Student En- 

chose the lowest rating of 

untrue gagement via Use of Technology 

Of the 27 responding administrators, 70% felt it was true (choice 4 and 5) that 
they observed increased student engagement. While just over 25% indicated 
neutrality by choosing 3 (interpreted as neither true nor untrue). 
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Open-ended items 

Similar to the teacher-colleague surveys, school administrators were asked to respond 
to three essay-style, open-ended questions to provide qualitative data for this study. School 
administrators were given the opportunity to describe in their own words what they perceived 
the role of a SETIS candidate to be, as well as what they hoped to accomplish working with a 
SETIS (administrator preprogram survey) and what they actually accomplished working with 
a SETIS (administrator postprogram survey). The final open-ended question gave school ad- 
ministrators the option to provide any additional comments they had about the SETIS pro- 
gram. 

Among school administrators who provided responses in both the pre- and postpro- 
gram surveys, we detected changes in attitudes and beliefs regarding both the SETIS candi- 
dates and SETIS program. We assessed comments in response to the first open-ended item 
for any shifts in school administrators’ perception of the role of the SETIS candidates. Of the 
16 paired pre-post responses, to indicated no change in their view of the SETIS role and the 
remaining six described more specific roles when comparing pre-to-post responses. 

The second open-ended item asked school administrators what they hoped to accom- 
plish by working with a SETIS candidate (preprogram) and what they actually did accomplish 
(postprogram). Once more, analyzing only matched pre-post survey responses (n = 16) we 
found that six school administrators believed their SETIS candidate met expectations, four 
described how their SETIS candidate exceeded them, five reported their SETIS candidate 
made other unexpected accomplishments, and one felt his or her SETIS candidate did not 
meet expectations. Of note, among administrators that reported their SETIS candidate met 
other unexpected accomplishments, most of their preprogram survey expectations involved 
improvements in student achievement including improved test scores, while their postpro- 
gram survey accomplishments often included increased technology integration and improved 
student engagement. 

The final open-ended item provided school administrators the opportunity to share 
any additional comments they had about the SETIS program. Of the matched pairs (n = 4) all 
comments were positive. Most preprogram survey comments expressed enthusiasm about uti- 
lizing the SETIS candidates to integrate technology at the student, teacher, and curriculum 
levels; with the postprogram comments typically praising the SETIS candidates and program. 
The unmatched pre- and postprogram survey comments were mostly positive in nature. How- 
ever, in one preprogram survey comment, a school administrator expressed a desire for a de- 
scription of what duties and services a typical SETIS performs. 


Discussion 

This study examined survey results from the three primary stakeholder groups in the 
Special Education Technology Integration Specialist (SETIS) program: SETIS candidates, 
teacher colleagues, and school administrators. The results enable us to address each of the 
evaluation questions chosen to measure the implementation, use, and impact of the SETIS 
program. 
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Evaluation Question 1 

Evaluation question one— To what extent does the TIS program build the capacity of 
participating TISs to plan and facilitate (a) teaching and learning, (b) information access 
and delivery, and (c) program administration?— is primarily addressed through the SETIS 
survey capacity indices with supporting evidence from other survey items. 

Improved capacity of SETIS candidates to plan and facilitate teaching and learning is 
evidenced by the organizational and human capacity indices and two other survey items. Or- 
ganizational capacity survey items assessed the ability of SETIS candidates to interact, com- 
municate, and collaborate with others (i.e., teach). This index demonstrated statistically 
significant median differences in retrospective pre-post survey item pairs. Further, over 70% 
of the SETIS candidates indicated they had conducted formal staff development related to 
technology integration at their school within the past year. The human capacity index, meas- 
uring the intellectual proficiency and will of the SETIS candidates, provided support for 
gained capacity among SETIS candidates to plan and facilitate learning. This index also re- 
vealed statistically significant shifts in retrospective pre-post responses, exhibiting a very large 
effect size. Additional evidence of enhanced learning among SETIS candidates can be drawn 
from the open-ended survey item which asked SETIS candidates to name the most valuable 
aspect or aspects of the SETIS program. Over half of the responding candidates (17 of 29) cited 
learning about technology resources/tools as the most valuable aspect of the program. 

The material capacity and organizational capacity indices provided a measure for the 
second part of EQi— that is, perceived increases in SETIS candidates’ capacity to plan and 
facilitate information access and delivery. Resource access, development, and sharing were 
each covered in the material capacity index. The material capacity index produced statistically 
significant median differences in retrospective pre-post responses, and exhibited a very large 
effect size. Survey items in the organizational capacity index provide additional validation that 
SETIS candidates experienced enhanced capacity in delivering technology integration to their 
schools. Additionally, when naming the most valuable aspect of the SETIS program, several 
individuals identified collaborating with other teachers/professional learning communities 
(seven of 29) and practice using new technology tools (five of 29) as the most valuable com- 
ponent of the program. 

The structural capacity index examined the final component to EQi, SETIS candidates 
capacity to plan and facilitate program administration. For the purposes of this survey, the 
items making up the structural capacity index concentrated on policies and practices SETIS 
candidates observed at their schools and with their teacher colleagues. While SETIS candi- 
dates do not directly change policies, procedures, and formalized practices at their schools, 
positive impact on these elements may be evident if SETIS candidates are capable of success- 
ful program administration. Within the structural capacity index, median differences in ret- 
rospective pre-post item responses were not as large as other indices. However, all items in 
this index remained statistically significant. The overall effect size for the index was medium. 
These results indicate SETIS candidates are having a positive and meaningful impact on struc- 
tures within the schools they support. 
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On the whole, the SETIS cohorts perceived substantial increases in their capacity to 
plan and facilitate all of the components in EQi. Each capacity index (human, material, or- 
ganizational, and structural) yielded statistically significant increases in the median differ- 
ences in retrospective pre-post survey results. Additionally, effect sizes for all indices, with the 
exception of structural, are considered very large (see Limitations of the Study for a discussion 
on the magnitude of effect sizes in relation to retrospective surveys). The strength of these 
results provides reasonable evidence that the SETIS program equipped its candidates with the 
capacity necessary to implement technology integration techniques within their schools and 
classrooms. 


Evaluation Question 2 

To answer the second evaluation question— To what extent do TISs encounter barriers 
to successful program implementation (e.g., financial, temporal, relational, etc.)?— we look 
at findings from the SETIS candidate and school administrator surveys. 

Multiple items from the SETIS survey address this evaluation question. First, as part 
of the material capacity index, SETIS candidates indicated their access to high quality tech- 
nology resources (e.g. computers, digital cameras, and whiteboards). Results for this item sug- 
gested that access to materials improved. Next, responses to some of the open-ended items 
shed light on perceived barriers faced by the SETIS candidates. When asked what aspects of 
the program they would change, many SETIS candidates suggested changes such as pre-set 
and specific deadlines on assignments, clearer expectations, and increased program organi- 
zation. Others stated they would like to see more face-to-face meetings. Of note, several SETIS 
candidates indicated they would not change any aspect of the program. While the responses 
to this question vary, they are similar to previous SETIS evaluation studies. In other words, 
most SETIS candidates would like a program with amplified accountability and additional in- 
person meetings. 

One barrier highlighted by an open-ended survey item was the perceived lack of school 
structures that support the SETIS program and teacher collaboration. Of the 29 responding 
SETIS candidates, eight felt there were no supports offered by their school. Finally, the last 
open-ended item asked SETIS candidates to list any barriers they may have encountered when 
implementing what they learned as part of the SETIS program. Of the seven themes identified 
during analysis, a shortage of time was the most common barrier (nine of 29), followed by 
receptiveness of teacher colleagues and school administrators (seven of 29), access to and 
quality of technology (five of 29), and none (five of 29). 

In previous SETIS evaluation studies, one barrier identified was a lack of awareness of 
the SETIS program and SETIS candidates among school administrators. However, this study’s 
results show over 90% of responding school administrators were aware they would have a 
SETIS in their school that year. This suggests program adjustments were adopted from prior 
evaluation study recommendations, resulting in improved communication among SETIS can- 
didates and their school administrators. 

Overall, the SETIS program throughout the 2012 to 2014 school years encountered 
relatively moderate barriers. From the perspective of the SETIS candidates, a lack of time was 
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their largest barrier followed closely by unreceptiveness of teacher colleagues and / or school 
administrators. Access to technology tools among SETIS candidates was not an issue, how- 
ever, computer access for students and internet speed was a concern. The SETIS candidates 
also identified several things they would change about the program; specifically deadlines on 
assignments, clearer expectations, and increased program organization. While these may or 
may not be barriers to program implementation, they should be taken under consideration. 
Finally, the previous barrier of inadequate awareness about the SETIS and SETIS program 
among school administrators seems to have been alleviated by programmatic adjustments. 

Evaluation Question 3 

Findings from the SETIS teacher colleague and school administrator pre- and post- 
program surveys were used to address the third evaluation question— To what extent is the 
level of technology integration in TIS schools positively impacted through participation in 
the program? 

When measuring positive impacts of school participation in the SETIS program, open- 
ended questions from the teacher-colleague pre- and postprogram surveys provided valuable 
information. First, when teacher colleagues were asked to describe the role of their SETIS 
candidate in the postprogram survey, many teachers articulated more specific candidate roles. 
In fact, most of these responses included descriptions of ways in which the SETIS candidate 
helped them integrate technology into their classrooms and curriculum. Next, looking at what 
teacher colleagues hoped to accomplish with their SETIS candidate (preprogram survey) and 
what they actually did accomplish (postprogram survey), we see that nearly half of the teacher 
colleagues felt their candidate met expectations. Further, two teachers indicated their SETIS 
exceeded expectations and five reported their SETIS met other, unexpected accomplishments. 
Again, the majority of the comments described technology integration in the classroom and/ or 
curriculum. 

The technology use indices from the teacher-colleague survey were also designed to 
help answer EQ3. While each index demonstrated an increase in mean score from pre to post- 
program survey data, Wilcoxon signed-rank tests revealed statistically significant results for 
the design and model indices only. These results suggest positive improvement in the abilities 
of teacher colleagues to more thoughtfully plan and implement technical assistance and re- 
lated activities (design index), as well as increased capacity to model digital-age work and 
learning techniques and behaviors (model index). 

Additional support of positive impact via technology integration at participating 
schools is found in the school administrator postprogram survey. When asked if they had ob- 
served increased student engagement in curricular activities by way of increased technology 
use attributable to the presence of a SETIS, 70% of school administrators indicated they had. 
Moreover, in open-ended survey responses administrators frequently noted increased tech- 
nology integration and use by teachers and students when describing the role of their SETIS 
candidate and the actual accomplishments achieved through working with them during the 
school year. 
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SETIS candidate activities that led to positive impacts in classrooms were associated 
with improved technology integration in classrooms and curriculum, raised technology 
knowledge among teachers, and enhanced student experiences. Perhaps most importantly, 
open-ended survey responses from both the teacher colleague and school administrators evi- 
denced the extent to which the SETIS candidates integrated technology in their schools. Fur- 
ther, data analysis indicates improved technology use among teacher colleagues in the design 
and model indices. 


Evaluation Question 4 

The fourth evaluation question— In what ways have school administrators and teach- 
ers leveraged the TIS and the resources provided by the TIS?— was addressed using many of 
the same findings discussed in EQ3, in addition to items from the SETIS candidate survey. 

As discussed in EQ3, teachers and administrators reported increased technology use 
in classrooms (with teachers also citing improved technology knowledge) thus supporting the 
leveraging of SETIS candidates and resources. Data supporting this is found in statistically 
significant median differences between pairs of observations concerning technology use 
among teacher colleagues in the design and model indices. Utilization of the SETIS program 
may also be seen in the shift in perception of the SETIS role by both teacher colleagues and 
school administrators. Among both stakeholder groups, several postprogram survey re- 
sponses were more specific in the role description, with the teachers and administrators often 
citing examples of how they had worked with their SETIS during the school year. Additionally, 
70% of school administrators reported observing increased student engagement as a result of 
technology integration brought about by their SETISs’ activities and efforts. 

Further evidence of leverage can be detected in open-ended survey items focused on 
expected and actual use of SETIS candidates and resources. As considered in the EQ3 discus- 
sion, a number of teacher colleagues reported their SETIS candidates met or exceeded their 
expectations throughout the school year and others reported that their SETIS candidates met 
other unexpected accomplishments. This was true for school administrators as well; they re- 
ported their SETIS candidates met or exceeded expectations and/or met other unexpected 
accomplishments. Contrary to results from these open-ended responses, data analysis of the 
multiple choice survey items measuring the intention among school administrators to utilize 
their SETIS candidates paints a different picture. Preprogram survey data show high levels of 
intent in utilizing the SETIS candidates. However, postprogram survey results indicate ad- 
ministrators did not leverage their SETIS candidate to the extent they had originally planned. 
In fact, Wilcoxon signed-rank tests revealed statistically significant differences from the pre- 
program survey intention to the postprogram survey actual utilization of the candidates in a 
negative direction by school administrators among all items. 

Open-ended survey responses from the SETIS candidates provide additional examples 
illustrating both strengths and weaknesses when it comes to leveraging candidates and the 
resources they bring to their schools and classrooms. First, when asked to discuss the ways 
they worked with the administration at their school to address technology-related issues or 
concerns throughout the school year, the most common themes were providing specific assis- 
tance, discussing resource/technology needs, and other. Notably, only two SETIS candidates 
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indicated that they had not worked with their administrator to address school-level technol- 
ogy-related issues or concerns. Second, when prompted to name school structures that sup- 
ported the TIS program and teacher collaboration, the most commonly named support was 
administrative, followed by none, common planning time, and professional learning commu- 
nities. 

Collectively, these data suggest some school administrators and teacher colleagues are 
more fully leveraging their SETIS candidates than others. Wilcoxon signed-rank tests revealed 
positive increases in technology use among teacher colleagues with statistically significant 
growth in the design and model indices. On the other hand, Wilcoxon signed-rank tests re- 
sulted in statistically significant reductions when comparing school administrators prepro- 
gram survey intentions to the postprogram survey utilization of their SETIS candidates. Yet, 
despite these data, open-ended survey results among the teacher colleagues and school ad- 
ministrators portrayed a high level of satisfaction with the accomplishments achieved by 
working with a SETIS candidate. 


Evaluation Question 5 

The fifth and final evaluation question— What impact has the TIS program had on 
students’ technology literacy in participating schools?— relies upon the premise that success- 
ful program implementation along with efficient SETIS and technology resource use leads to 
higher teacher, administrator, and student technology literacy levels. To address EQ5 we draw 
insight from the teacher colleague and school administrator pre- and postprogram surveys. 

Qualitative open-ended survey items from both surveys provided the most compelling 
evidence to support the premise. When asked about accomplishments and outcomes from 
participation in the SETIS program, a number of teacher colleagues and school administrators 
cited increased use of technology in classrooms/schools as an accomplishment. Multiple 
teacher colleagues described postprogram enhanced student experiences as the most notable 
accomplishment. Additionally, 70% of school administrators reported observing increased 
student engagement as a result of technology integration brought about through SETIS activ- 
ities and efforts. Interestingly, in the preprogram survey it was common for administrators to 
name improved student achievement as an expected accomplishment and then to cite in- 
creased technology integration and student engagement as the actual accomplishment. From 
these open-ended remarks, it is evident that both teachers and administrators observed not 
only increased technology use in classrooms, but also enriched student experiences. 

All technology use indices (teacher colleague pre- and postprogram surveys), with the 
exception of the engage index, contain items with reference to students. Examined as individ- 
ual items, only a few items revealed statistically significant increases in mean survey scores 
from pre- to postprogram surveys. However, Wilcoxon signed-rank tests revealed positive 
increases in technology use among teacher colleagues with statistically significant growth in 
the design and model indices. As discussed in EQ3, these results suggest positive shifts in the 
abilities of teacher colleagues to thoughtfully plan and implement technical-assistance-related 
activities (design index), as well as improved capacity to model digital-age work and learning 
techniques and behaviors (model index). 
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Survey results give evidence that both teacher colleagues and school administrators 
perceived a rise of technology integration in classrooms and curriculum. Further, the majority 
of school administrators reported they had observed increased student engagement as a result 
of technology integration efforts by their SETIS. Analysis of the teacher colleague technology 
use indices showed statistically significant increases in the design and model indices. This 
shift in behavior indicates that, as a result of the SETIS program, teacher colleagues who re- 
sponded to the survey are practicing technology integration techniques and engaging their 
students using technology resources provided by their SETIS. 

Limitations of the Study 

When interpreting results in any study, it is important to consider inherent limitations 
that may skew findings. While sound research methods are an integral component of high 
quality studies, eliminating all potential risks of bias is impossible. The limitations of this eval- 
uation study are typical of other similar studies as discussed below. 

Surveys that rely upon self-reported information always have a risk of response bias; 
respondents may exaggerate or underestimate, may have accurate recall difficulties, and may 
report information they perceive as socially acceptable. This study relies upon two types of 
self-reported surveys: a retrospective pre-post survey and two traditional pre-post surveys. 

Previous SETIS program evaluation studies exhibited a limitation of small sample 
sizes. This study, in part due to combining the cohorts from the 2012-2013 and 2013-2014 
school years, does not have this limitation. The SETIS candidate retrospective survey was 
completed by 31 SETIS candidates, while 51 teacher colleagues responded to both the teacher 
colleague pre- and postprogram surveys and 33 school administrators participated in both the 
school administrator pre- and postprogram surveys. Therefore, the sample sizes of all three 
stakeholder groups are adequate for us to draw general conclusions about the groups with a 
certain level of confidence. 

Determining response rates among teacher colleagues and school administrators re- 
mains impossible due to the fact that the SETIS candidates invite an unknown number of 
teachers and administrators to participate in the pre- and postprogram surveys. However, a 
key limitation in previous SETIS program studies— the inability to match pre- and postpro- 
gram survey results among the teacher colleagues and school administrators— has been ad- 
dressed by adjusting the study’s methodology. With the incorporation of a mechanism which 
enables matching individual pre- and postprogram survey responses, it is possible to measure 
with more confidence shifts in knowledge, skills, and behaviors at individual and group levels. 

Retrospective pre- and postprogram surveys are convenient because they occur once 
(improving response rates) and the pre/post data are matched (the same participants re- 
sponding to the preprogram survey also responded to the postprogram survey). Certain re- 
search postulates that a form of bias called response-shift bias is alleviated through using 
retrospective pre- and postprogram surveys. Moore and Tananis (2009) define response-shift 
bias as occurring “when a participant uses a different internal understanding of the construct 
being measured to complete the pretest and posttest.” Simply put, it is common for partici- 
pants to overestimate their knowledge, skills, and behaviors before exposure to a program or 
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intervention. However, after the completion of a program, (and caused by what they learned 
in the program) they may realize their knowledge, skills, and behaviors were not truly as high 
as they first estimated. Therefore, it is possible for participants to rate themselves lower in a 
posttest due to a shift in their frame of reference, potentially masking the actual impact of a 
program. Conversely, other research argues traditional pretest/posttest types of surveys result 
in less biased program effectiveness estimates. In a 2011 study, Nimon, Zigarmi, and Allen 
discovered inflated effect sizes among retrospective pre- and postprogram survey items. Ac- 
cording to their findings, surveys with before and after items presented side by side may in- 
troduce types of bias including theories of change, self-presentation, and/or effort 
justification. Further, to resolve the issues of both response-shift bias and exaggerated effect 
sizes Nimon et al. (2011) recommend administering traditional pre- and postprogram surveys 
with a retrospective pre- and postprogram survey. While this solution would alleviate some 
bias, it was not possible to initiate in this study. 

Recommendations 

In general, survey data from all three SETIS program stakeholders suggest the 2012 
through 2014 school years’ SETIS program cohorts were mostly successful. The greatest im- 
pact occurred among the SETIS candidates within the four indices of capacity building— hu- 
man, material, organizational, and structural— all yielding statistically significant and 
practically important increases from pre-to postprogram survey observations. Among teacher 
colleagues, the use of Wilcoxon signed-rank tests revealed statistically significant median dif- 
ferences between pairs of observations in technology use for the model and design indices. 
While Wilcoxon signed-rank tests also resulted in statistically significant differences between 
the school administrator pre- and postprogram surveys, the differences uncovered a disparity 
between the preprogram survey intentions to utilize SETIS candidates and postprogram sur- 
vey responses about their actual utilization by school administrators. Despite lower-than-ex- 
pected leveraging of SETIS resources, other survey data indicated successes resulting from the 
SETIS program. 

Study results also suggest that previous evaluation study recommendations have been 
adopted by the SETIS program. These changes include methodology adjustments to allow pre- 
and postprogram survey matching among the teacher colleague and school administrator 
stakeholders as well as improved awareness among SETIS candidates and school administra- 
tors. Even with these promising program adjustments, there are several recommendations 
that may further improve the implementation, utilization, and impact of the SETIS program: 

• If logistically possible, consider holding more face-to-face meetings for the SETIS can- 
didate group only. In consideration of the specialized nature of special education con- 
tent, providing SETIS candidates greater opportunities to work together may help 
them more effectively implement technology integration in classrooms to benefit stu- 
dents with disabilities in particular. 

• At the school level, promote scheduling that allows teacher colleagues and SETIS can- 
didates time to cocreate technology-integrated lesson plans. Given time to thought- 
fully plan and collaborate, teachers may be more inclined to incorporate the 
technological tools and resources the SETIS candidates offer. 
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• Encourage further collaboration between SETIS candidates and their school adminis- 
trators to assist administrators in leveraging the SETIS candidates to a fuller extent. 
With a significant disparity between the preprogram survey intentions and the post- 
program survey actual utilization of the candidates by school administrators, there is 
ample opportunity to improve leveraging SETIS candidates and the resources they 
bring to their schools. 
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Appendix A. SETIS Candidate Retrospective Pre- and 
Postprogram Survey Results 


Table 1. Human Capacity Items: Retrospective SETIS Candidate Survey Pre- and Postprogram Means, 
Wilcoxon Signed-Rank Test and Effect Size 


Pre 


Item Item statement 


n Mean SD 


Significance of 

Post difference Cohen's Effect 

n Mean SD z p d size 


1 

1 have a strong understanding of 
how to use/apply the resources 
from Thinkfinity. 

31 

2.65 

0.95 

31 

4.23 

0.56 

-4.83 

<.001 

2.06 

very 

large 

2 

1 have a strong understanding of 
how to use/apply the resources 
from Curriculum Pathways (SAS). 

31 

2.61 

1.05 

31 

4.26 

0.82 

-4.74 

<.001 

1.77 

very 

large 

3 

1 have a strong understanding of 
how to use/apply the resources 
from West Virginia Writes. 

31 

3.23 

1.06 

30 

4.10 

0.80 

-4.20 

<.001 

0.95 

large 

4 

1 have a strong understanding of 
how to use/apply the resources 
from Acuity. 

31 

3.29 

1.04 

31 

4.26 

0.77 

-4.28 

<.001 

1.07 

large 

5 

1 have a strong understanding of 
how to use/apply the resources 
from TechSteps. 

31 

3.00 

1.06 

31 

4.26 

0.58 

-4.49 

<.001 

1.49 

very 

large 

6 

1 am able to assess the quality and 
legitimacy of web resources. 

31 

3.16 

1.04 

31 

4.61 

0.50 

-4.57 

<.001 

1.82 

very 

large 

7 

1 understand the most important 
issues surrounding legal 

31 

2.58 

1.03 

31 

4.45 

0.57 

-4.86 

<.001 

2.29 

very 

large 


10 


use/copyright regulations and 
how they relate to integrating 
web resources and technology 
into lesson plans and instruction. 

I am able to identify the 
components of a URL and to 
ensure it is legitimate (e.g., 
protocol, host, domain, directory, 
port address, etc.). 

I have a strong understanding of 
the core National Educational 
Technology Standards and 
Performance Indicators for 
Teachers (NETS-T). 

I have a strong understanding of 
the core National Educational 
Technology Standards and 
Performance Indicators for 
Students (NETS-S). 


31 2.42 0.85 31 4.13 0.62 


-4.78 <.001 


30 2.43 1.01 30 4.33 0.55 


-4.80 <.001 


28 2.43 1.00 29 4.28 0.45 


2.34 very 
large 


2.39 very 
large 


-4.63 <.001 2.44 very 

large 


Table 1 continues next page 
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Table 1. Human Capacity Items: Retrospective SETIS Candidate Survey Pre- and Postprogram Means, 


Wilcoxon Signed-Rank Test and Effect Size 


Item 

Item statement 

n 

Pre 

Mean 

SD 

n 

Post 

Mean SD 

Significance of 
difference 

z P 

Cohen's 

d 

Effect 

size 

11 

1 understand how to integrate 

Web 2.0 tools into instruction 
(e.g., podcasting, wikis and blogs, 
social networking, etc.). 

30 

2.63 

1.00 

30 

4.70 

0.47 

-4.78 

<.001 

2.70 

very 

large 

12 

1 integrate digital resources/tools 
into my work with teachers, 
students and administrators. 

31 

3.16 

0.90 

31 

4.71 

0.53 

-4.80 

<.001 

2.14 

very 

large 

13 

1 understand how to effectively 
integrate technology into 
instruction to improve the quality 
of students' educational 
experiences. 

30 

3.27 

0.91 

30 

4.70 

0.47 

-4.65 

<.001 

2.02 

very 

large 

14 

1 have a strong understanding of 
the county/school acceptable use 
policy. 

30 

3.50 

0.97 

30 

4.50 

0.57 

-4.28 

<.001 

1.27 

large 

15 

1 have a strong understanding of 
21st century assessment. 

31 

3.06 

0.96 

31 

4.52 

0.51 

-4.64 

<.001 

1.92 

very 

large 

16 

1 have a strong understanding of 
how to design and implement 
project-based learning (PBL) in 
the classroom. 

31 

3.13 

0.92 

31 

4.48 

0.51 

-4.66 

<.001 

1.85 

very 

large 

17 

1 actively use action research to 
assess the impact of my teaching 
on student learning. 

31 

2.71 

1.07 

31 

4.35 

0.55 

-4.52 

<.001 

1.96 

very 

large 


Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

Scale for means: 5-point Likert-type scale, ranging from a low of 1 (strongly disagree) to a high of 5 (strongly agree) 
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Table 2. Organizational Capacity Items: Retrospective SETIS Candidate Survey Pre- and Postprogram 
Means, Wilcoxon Signed-Rank Test and Effect Size 


Item Item statement 

18 I model lessons that 
integrate technology for 
other teachers in the school. 

19 I collaborate with others 
within the school to 
effectively integrate 
technology into instruction. 

20 I serve as a resource to 
other teachers regarding the 
effective use of technology 
in the school. 

21 I can effectively work with 
others to assess their 
learning and information 
needs (e.g., other teachers, 
students, administrators, 
etc.). 

22 I have an ongoing dialogue 
with other staff members at 
the school about technology 
issues and how they can be 
addressed. 


Pre 

n Mean SD 

31 2.77 0.92 


31 2.94 1.03 


31 3.29 0.94 


31 3.48 0.81 


31 3.13 0.96 


Significance of 

Post difference Cohen's Effect 


Mean 

SD 

Z 

P 

d 

size 

4.13 

0.76 

-4.76 

<.001 

1.63 

very 

large 

4.52 

0.63 

-4.81 

<.001 

1.88 

very 

large 

4.65 

0.61 

-5.00 

<.001 

1.74 

very 

large 

4.65 

0.49 

-4.88 

<.001 

1.77 

very 

large 

4.35 

0.66 

-4.52 

<.001 

1.52 

very 

large 


Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

Scale for means: 5-point Likert-type scale, ranging from a low of 1 (strongly disagree) to a high of 5 (strongly 

agree) 
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Table 3. Structural Capacity Items: Retrospective SETIS Candidate Survey Pre- and Postprogram Means, 
Wilcoxon Signed-Rank Test and Effect Size 

Significance of 





Pre 



Post 

difference 

Cohen's 

Effect 

Item 

Item statement 

n 

Mean 

SD 

n 

Mean 

SD 

z 

P 

d 

size 

23 

The school has set a time and 
place where staff meet in pro- 
fessional communities of prac- 
tice to discuss how to 
effectively integrate technology 
into instruction. 

31 

2.87 

1.02 

31 

3.71 

0.94 

-4.13 

<.001 

0.87 

large 

24 

Teachers at the school have 
time to co-plan and/or co-teach 
lessons that integrate technol- 
ogy into instruction. 

31 

2.84 

1.04 

31 

3.45 

0.77 

-2.93 

0.003 

0.68 

medium 

25 

Staff at the school understand 
the acceptable use policy. 

31 

3.48 

0.89 

31 

4.10 

0.70 

-3.76 

<.001 

0.78 

medium 

26 

Staff at the school understand 
key concepts and best practices 
regarding web literacy. 

31 

2.90 

0.91 

31 

3.68 

0.75 

-3.90 

<.001 

0.95 

large 

27 

Staff at the school understand 

31 

2.74 

0.82 

30 

3.60 

0.86 

-4.13 

<.001 

1.04 

large 


the most important issues sur- 
rounding legal use/copyright 
regulations and how they relate 
to integrating web resources 
and technology into lesson 
plans and instruction. 


28 

Staff at the school are familiar 
with the technology compo- 
nents of the school's strategic 
plan. 

30 

2.80 

0.89 

30 

3.63 

0.96 

-3.99 

<.001 

0.92 

large 

29 

The school has policies and pro- 
cedures in place that support 
the use of Project Based Learn- 
ing (PBL). 

31 

3.26 

0.96 

31 

3.77 

1.02 

-3.23 

0.001 

0.53 

medium 

30 

The school has policies and pro- 
cedures in place that support 
the use of technology resources 
(e.g., Thinkfinity, Acuity, West 
Virginia Writes, etc.). 

31 

3.77 

0.88 

31 

4.16 

0.64 

-2.97 

0.003 

0.51 

medium 

31 

Staff at the school regularly use 
Thinkfinity and or Curriculum 
Pathways (SAS) as a resource in 
the classroom. 

31 

3.00 

1.06 

30 

3.40 

1.10 

-2.81 

0.005 

0.38 

small 









Table 3 continues next page 

32 

Staff at the school regularly use 

31 

3.87 

0.81 

31 

4.19 

0.70 

-3.16 

0.002 

0.43 

small 


West Virginia Writes or another 
online writing program as a 
formative assessment of stu- 
dent writing. 
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Table 3. Structural Capacity Items: Retrospective SETIS Candidate Survey Pre- and Postprogram Means, 
Wilcoxon Signed-Rank Test and Effect Size 


Significance of 

Pre Post difference C ohen's Effect 


Item 

Item statement 

n 

Mean 

SD 

n 

Mean 

SD 

Z 

P 

d 

size 

33 

The school has a plan in place 
to support the implementation 
of TechSteps. 

31 

3.58 

1.09 

31 

4.16 

0.86 

-3.45 

0.001 

0.6 

medium 

34 

Staff at the school regularly use 
TechSteps as part of their core 
content instruction. 

31 

2.97 

1.17 

31 

3.39 

1.20 

-2.75 

0.006 

0.36 

small 

35 

Staff at the school understand 

the core National Education 
Technology Standards and 
Performance Indicators for 
Teachers (NETS-T). 

30 

2.80 

1.06 

31 

3.29 

0.97 

-3.42 

0.001 

0.49 

small 

36 

Staff at the school understand 

the National Education 
Technology Standards and 
Performance Indicators for 
Students (NETS-S). 

31 

2.84 

1.00 

30 

3.37 

1.00 

-3.64 

<.001 

0.54 

medium 

37 

Staff at the school understand 
West Virginia's 21st Century 

Skills and Tools. 

31 

3.19 

0.87 

31 

3.90 

0.79 

-3.95 

<.001 

0.87 

large 

38 

Staff at the school use Web 2.0 

tools for collaboration and 

instruction. 

31 

2.74 

0.89 

30 

3.73 

0.83 

-4.15 

<.001 

1.17 

large 

39 

Staff at the school frequently 

31 

3.16 

1.00 

31 

4.06 

0.63 

-3.82 

<.001 

1.1 

large 


integrate digital 
resources/tools in their 
teaching. 

Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

Scale for means: 5-point Likert-type scale, ranging from a low of 1 (strongly disagree) to a high of 5 (strongly agree) 
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Table 4. Material Capacity Items: Retrospective SETIS Candidate Survey Pre- and Postprogram Means, 
Wilcoxon Signed-Rank Test and Effect Size 


Pre 


Item Item statement n Mean 


40 


41 


42 


43 


44 


I have access to a variety of 31 
high quality technology 
resources at my school 
(e.g., computer, digital 
camera, whiteboard, etc.). 

I participated in a variety of 31 
professional development 
about integrating 
technology resources into 
instruction. 

I feel I am able to direct 31 
staff in my school toward 
high quality technology 
resources that are relevant 
to their information needs. 

I have the ability to develop 31 
useful technology resources 
for my school that address 
our information needs. 

I know where to find useful 31 
and high quality technology 
resources that can be 
integrated into instruction. 


3.77 


3.48 


2.87 


3.13 


3.19 


SD 

1.15 


1.09 


1.06 


1.02 


1.05 


Significance of 

Post difference Cohen's Effect 


Mean 

SD 

Z 

P 

d 

size 

4.65 

0.80 

-4.01 

<.001 

0.9 

large 

4.58 

0.67 

-3.89 

<.001 

1.23 

large 

4.45 

0.51 

-4.62 

<.001 

1.94 

very 

large 

4.52 

0.51 

-4.65 

<.001 

1.74 

very 

large 

4.71 

0.46 

-4.72 

<.001 

1.91 

very 

large 


Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

Scale for means: 5-point Likert-type scale, ranging from a low of 1 (strongly disagree) to a high of 5 (strongly 

agree) 


Table 5. Standardized Capacity Indices: Retrospective SETIS Candidate Survey Pre- and Postprogram 
Means, Wilcoxon Signed-Rank Test and Effect Size 

Significance of 

Pre Post difference 


Index 

n 

Mean 

SD 

Mean 

SD 

Z 

P 

Cohen's d 

Effect size 

Human 

capacity 

31 

2.97 

0.91 

4.44 

0.62 

-18.82 

<.001 

1.94 

very large 

Organizational 

capacity 

31 

3.17 

0.83 

4.50 

0.50 

-10.64 

<.001 

1.96 

very large 

Structural 

capacity 

31 

3.06 

0.96 

3.74 

0.99 

-14.50 

<.001 

0.7 

medium 

Material 

capacity 

31 

3.19 

1.00 

4.63 

0.49 

-9.71 

<.001 

1.87 

very large 


Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

Scale for means: 5-point Likert-type scale, ranging from a low of 1 (strongly disagree) to a high of 5 (strongly 

agree) 
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Table 6. Summarized Open-Ended Survey Items, Retrospective SETIS Candidate Survey: Valuable 
Aspects of SETIS Program Participation 

Frequency of comments* 


Theme SETIS survey ( n = 29) 

Learning about technology resources/tools 17 

Collaborating with other teachers /professional learning 

communities 7 

Gained qualification / confidence 2 

Face-to-face meetings / hands-on training 2 

Using specific technology tools 2 

Practice using new technology tools 5 

Implementing technology into classrooms 3 


Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

*Some respondents provided more than one comment; therefore, the frequency of comments may exceed 

the number of respondents (n). 


Table 7. Summarized Open-Ended Survey Items, Retrospective SETIS Candidate Survey: Suggested 
Changes to the SETIS Program 


Theme 

Frequency of comments 

SETIS survey ( n = 28) 

Program/content change 

13 

Time change 

3 

Meeting change 

5 

No change 

6 

N/A 

1 

Data source: SETIS candidate retrospective surveys, 2012- 

-2013 and 2013-2014 cohorts 


Table 8. Summarized Open-Ended Survey Items, Retrospective SETIS Candidate Survey: School 
Structures Supporting SETIS and Teacher Collaboration 


Theme 


Frequency of comments* 
SETIS survey (n = 29) 


Administrative support 
Staff support 
Common planning time 
PLC 

TIS support 

None 

Other 


10 

4 

7 
6 
1 

8 
2 


Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

*Some respondents provided more than one comment, therefore, the frequency of comments may exceed 

the number of respondents (n). 
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Table 9. Summarized Open-Ended Survey Items, Retrospective SETIS Candidate Survey: Addressing 
Technology Issues/Concerns (SETIS Candidate Support to School Administration) 


Theme 

Frequency of comments* 

SETIS survey (n = 29) 

Provide training/ professional development 

5 

Membership in school leadership team 

2 

Discuss resource/technology needs 

5 

Provide specific assistance 

8 

Increase technology as a school goal 

1 

Other 

5 

None 

2 

N/A 

2 

Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

*Some respondents provided more than one comment, therefore, the frequency of comments may exceed 

the number of respondents (n). 


Table 10. Summarized Open-Ended Survey Items, Retrospective SETIS Candidate Survey: Barriers 
Encountered by SETIS Candidates 

Frequency of comments* 


Theme SETIS survey ( n = 29) 

Time 9 

Access to and quality of technology 5 

Receptiveness of teacher colleagues / school administrators 7 

Other duties/ responsibilities 2 

Funding 1 

None 5 

Other 3 


Data source: SETIS candidate retrospective surveys, 2012-2013 and 2013-2014 cohorts 

*Some respondents provided more than one comment, therefore, the frequency of comments may exceed 

the number of respondents (n). 
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Appendix B. Teacher Colleague Pre- and Postprogram Survey 
Results 


Table 11. Facilitate Items: Teacher Colleague Pre- and Postprogram Survey Means, Wilcoxon Signed- 
Rank Test 

Significance 

of 

Pre Post difference 


Item 

Item statement 

n 

Mean 

SD 

n 

Mean 

SD 

Z 

P 


1 use information from digital 

50 

3.14 

0.87 

51 

3.18 

0.74 

-0.35 

0.73 

1 

sources to promote learning and 
engage students in classroom 
activities. 

1 require my students to use digital 

50 

2.66 

0.88 

51 

2.65 

0.96 

-0.36 

0.72 

2 

resources and tools for writing, 
collaboration, reflection, research, 
and other assignments. 

1 engage my students in real-world 

49 

3.00 

0.74 

51 

3.04 

0.66 

-0.58 

0.56 

3 

issues and authentic problem- 
solving. 

1 require my students to gather 

50 

2.54 

0.87 

51 

2.61 

0.94 

-0.68 

0.50 

4 

information from sources other 

than their textbooks in order to 
complete their daily assignments 
(e.g., podcasts, videos, etc.). 

1 require my students to present 

50 

2.36 

0.97 

51 

2.39 

0.85 

-0.19 

0.85 


information and actively teach 
content to their fellow students 
and/or community members. 

Data source: Teacher colleague pre-post surveys, school years 2012-2013 and 2013-2014 

Scale for means: 4-point Likert type scale where 1 = not usually, 2 = sometimes, 3 = often, and 4 = almost always 

Note: Cohen's d effect sizes could not be computed for any individual item results because none approached statistical 

significance. 
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Appendix B. Teacher Colleague Pre- and Postprogram Survey Results 


Table 12. Design Items: Teacher Colleague Pre- and Postprogram Survey Means, Wilcoxon Signed-Rank 
Test 

Significance 

Pre Post of difference 

Item Item Statement n Mean SD n Mean SD z p 

1 I use a blend of both face-to-face and 50 2.30 1.04 50 2.46 1.02 -0.56 0.57 

online environments to deliver 

instruction to my students. 

2 I design and/or implement projects that 50 2.60 0.76 50 2.74 0.81 -1.25 0.21 

emphasize creative thinking and 

require students to engage in problem- 
solving, decision-making and 
experimental inquiry, using digital 
resources/tools when appropriate. 

3 I often design and/or utilize student- 50 2.78 0.86 50 2.84 1.05 -0.74 0.46 

centered formative and performance- 

based assessments using available 
digital resources and tools (e.g., WV 
Writes, Acuity, TechSteps, etc.). 

4 I use digital resources and tools to 50 2.50 0.86 49 2.69 0.87 -1.54 0.12 

make assignments for students that are 

based upon their individual interests, 
abilities and learning needs. 

5 I require my students to set personal 50 2.18 0.92 50 2.44 0.94 -1.56 0.12 

learning goals and to self-assess their 

progress toward meeting those goals, 
using digital resources and tools when 
available and appropriate. 

Data source: Teacher colleague pre-post surveys, school years 2012-2013 and 2013-2014 

Scale for means: 4-point Likert type scale where 1 = not usually, 2 = sometimes, 3 = often, and 4 = almost always 

Note: Cohen's d effect sizes could not be computed for any individual item results because none approached statistical 

significance. 
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Appendix B. Teacher Colleague Pre- and Postprogram Survey Results 


Table 13. Model Items: Teacher Colleague Pre- and Postprogram Survey Means, Wilcoxon Signed-Rank 
Test and Effect Size 


Significance 

Pre Post of difference Cohen's Effect 


Item 

Item statement 

n 

Mean 

SD 

n 

Mean 

SD 

Z 

P 

d size 

1 

1 use digital resources and 
tools to communicate with 

students. 

50 

2.28 

0.94 

50 

2.48 

1.11 

-0.89 

0.37 


2 

1 use digital resources and 
tools to communicate with 

50 

2.96 

0.92 

50 

3.32 

0.74 

-2.81 

0.01 

0.43 small 


my peers. 










3 

1 use digital resources and 
tools to communicate with 
parents and the community 
outside of my school. 

50 

2.66 

0.83 

50 

2.86 

0.90 

-1.28 

0.20 


4 

1 customize the available 

48 

2.42 

1.07 

50 

2.52 

1.05 

-0.87 

0.39 



digital resources and tools 
such as WV Writes, Acuity, 

TechSteps, etc. to 
personalize learning for my 
students. 

Data source: Teacher colleague pre-post surveys, school years 2012-2013 and 2013-2014 

Scale for means: 4-point Likert type scale where 1 = not usually, 2 = sometimes, 3 = often, and 4 = almost always 

Note: Cohen's d effect sizes were not computed for any individual item results that did not approach statistical 

significance. 
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Appendix B. Teacher Colleague Pre- and Postprogram Survey Results 


Table 14. Promote Items: Teacher Colleague Pre- and Postprogram Survey Means, Wilcoxon Signed-Rank 
Test 

Significance of 

Pre Post difference 

Item Item statement n Mean SD n Mean SD z p 

1 I advocate, model and teach my 49 3.27 0.84 49 3.27 0.81 -0.61 0.54 

students about safe, legal and ethical 

use of digital information and 
technology, including respect for 
copyright, intellectual property, and 
the appropriate documentation of 
sources. 

2 I offer students opportunities to use 49 2.16 1.07 49 2.45 1.15 -1.43 0.15 

digital resources and tools to 

participate in collaborative projects 
with students of other cultures that 
address current problems, issues or 
themes. 

3 Students in my class model 48 3.23 0.90 48 3.38 0.87 -0.55 0.58 

appropriate online behavior and social 

interaction through digital activities in 
my classroom. 

Data source: Teacher colleague pre-post surveys, school years 2012-2013 and 2013-2014 

Scale for means: 4-point Likert type scale where 1 = not usually, 2 = sometimes, 3 = often, and 4 = almost always 

Note: Cohen's d effect sizes could not be computed for any individual item results because none approached statistical 

significance. 
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Appendix B. Teacher Colleague Pre- and Postprogram Survey Results 


Table 15. Engage Items: Teacher Colleague Pre- and Postprogram Survey Means, Wilcoxon Signed-Rank 
Test and Effect Size 


Pre 

Item Item Statement n Mean SD 

1 I participate actively in local 49 2.65 0.99 

communities of practice with my 

fellow teachers, either online or 
face-to-face. 

2 I participate actively in 49 1.94 0.83 

online/global communities with 

other educators outside of my 
local community to gather and 
discuss resources and ideas re- 
lated to student achievement 
and learning. 

3 I model and teach other educa- 49 2.10 0.96 

tors to use digital tools and re- 
sources to promote student 

achievement and learning. 

4 I regularly seek out digital re- 49 2.88 0.86 

sources, tools and research and 

evaluate its quality and rele- 
vance prior to using it in the 
classroom. 


Significance 

Post of difference Cohen's Effect 

n Mean SD z p d size 

50 2.68 0.83 -0.25 0.80 


50 2.10 0.87 -0.69 0.49 


49 2.39 1.00 -1.96 0.05 0.29 small 


50 2.94 0.93 -0.35 0.73 


Data source: Teacher colleague pre-post surveys, school years 2012-2013 and 2013-2014 

Scale for means: 4-point Likert type scale where 1 = not usually, 2 = sometimes, 3 = often, and 4 = almost always 

Note: Cohen's d effect sizes were not computed for any individual item results that did not approach statistical 

significance. 


Table 16. Differences in Technology Use Indices: Teacher Colleague Pre- and Postprogram Survey 
Means, Wilcoxon Signed-Rank Test and Effect Size 


Index 


Pre 

Post 

Significance of 
difference 

Cohen's 

d 

Effect size 

n 

Mean 

SD 

Mean 

SD 

z 

P 

Facilitate 

51 

2.74 

0.90 

2.77 

0.88 

-0.56 

0.58 



Design 

51 

2.47 

0.90 

2.63 

0.95 

-2.39 

0.02 

0.18 

negligible 

Model 

51 

2.58 

0.97 

2.80 

1.01 

-2.80 

0.005 

0.22 

small 

Promote 

51 

2.88 

1.07 

3.03 

1.04 

-0.99 

0.32 



Engage 

51 

2.39 

0.98 

2.53 

0.95 

-1.12 

0.26 



Data source: 

Teacher colleague pre-post surveys, school years 2012-2013 and 2013- 

-2014 




Scale for means: 4-point Likert type scale where 1 = not usually, 2 = sometimes, 3 = often, and 4 = almost always 
Note: Cohen's d effect sizes were not computed for any index results that did not approach statistical significance. 
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Appendix C. School Administrator Pre- and Postprogram 
Survey Results 


Table 17. SETIS School Administrator Survey Items: Pre- and Postprogram Means, Wilcoxon Signed-Rank Test 
and Effect Size 


Pre 


Post 


Item Item statement 

Share I plan to ask my SETIS to share/ My 

SETIS shared what he/she has learned 
by leading professional development 
for the other teachers in my school. 
Model I plan to ask my SETIS to model/ My 
SETIS modeled the integration of 
technology for his/her coteachers 
and others within the school. 

Assist in I anticipate asking my SETIS to assist/ 
plan My SETIS assisted me in developing 

the school's strategic plan with 
regard to information and technology 
needs. 

Assist I expect that my SETIS will assist/ My 
teacher SETIS assisted his/her coteachers in 
customizing available digital 
resources and tools such as WV 
Writes, TechSteps, Acuity, Edmodo, 
Thinkfinity, etc. to personalize 
learning for students. 

Work I will request that the SETIS/ I 
with requested that the SETIS work with 
teacher teachers to identify digital resources 
and tools that effectively integrate 
technology into their current 
curriculum. 


n Mean SD n Mean SD 
33 4.55 0.62 26 3.96 0.96 


33 4.82 0.46 27 4.15 1.28 


32 3.91 1.03 26 3.62 1.10 


33 4.67 0.54 27 4.15 1.39 


33 4.58 0.56 24 3.63 1.13 


Significance 

of difference Cohen's Effect 
z p d size 
-2.39 0.017 0.75 medium 


-3.25 0.001 0.74 medium 


-2.00 0.045 0.28 small 


-2.84 0.004 0.52 medium 


-3.11 0.002 1.14 large 


Conduct I will ask the SETIS/ 1 asked the SETIS 33 4.27 0.88 27 3.22 1.38 -3.37 0.001 0.95 large 

analysis to conduct analyses of student data 
and engage in action research to help 
me understand the impact of 
technology integration at my school. 

Observe Based on your classroom NA NA NA 27 4.19 1.67 NA NA NA NA 

observations this school year, has the 
SETIS candidate increased student 
engagement in curricular activities 
through increased use of technology. 


Data source: School administrator pre-post surveys, school years 2012-2013 and 2013-2014 

Scale for means: Pre survey used 5-point scale of likelihood where the lowest rank of 1 = unlikely and highest rank of 5 = likely. 
Post survey used 5-point scale of agreement where the lowest rank of 1 = untrue and the highest rank of 5 = true. 
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